Your benchmark is invalid, you don't even show the baseline. FORCE_MMVQ enables it anywhere, which is not default behaviour. Also, here's a counter example on AMD Strix Halo:

before is just base master, after is with GGML_VK_DISABLE_MMVQ=1

Yes It was a very quick way to test. After testing with several model I found that the performance was not as prominent. It might due to the thermal behavior my laptop.

Benchmark on the patch

To isolate the behavior, I profiled the changes using a benchmark script that measures the performance impact of MMVQ (Matrix-Matrix-Vector Quantization) during matrix multiplication operations (MUL_MAT).

Below is the summary of the benchmark run on my AMD UMA device (using the maximum GFLOPS over 4 warm repeats, skipping 1 cold burn-in):

SUMMARY (max GFLOPS of 4 warm repeats, 1 cold burn-in skipped)

Key Observations:

1. Baseline Consistency: As expected, the f16 baselines are essentially identical across both modes since f16 does not trigger MMVQ paths.
2. Quantization Overhead at $K=4096$: The computational overhead of MMVQ is highly visible at smaller vector widths. Disabling MMVQ yields massive synthetic throughput gains (up to +57.7% for q6_K).
3. Diminishing Returns at Larger $K$: When $K$ expands to 14,336, the gaps narrow significantly. Disabling MMVQ does not trigger a performance regression here, meaning memory bandwidth savings from weight compression begin to balance out the shader arithmetic overhead.

Overall

Following the isolated benchmarks, I tested several real-world models to evaluate the end-to-end impact of the patch during single-token generation (tg128, not using flags):

Given that AMD UMA hardware can be highly sensitive to background noise, thermal behavior, and initial shader compilation overhead (especially if the Vulkan pipeline cache isn't warmed up, that also happen to me during the benchmark setup). Could you try running a re-test again?

And you think I'm not aware of thermal behaviour? llama-bench does multiple runs for that reason, at most you see a higher variance.

With your latest results you didn't even show whether you used FORCE_MMVQ or not.

Regardless, a blanket disable on UMA devices makes no sense, and you should really not be using claude to try to explain the reason for performance behaviour, it is guaranteed to hallucinate.

And you think I'm not aware of thermal behaviour? llama-bench does multiple runs for that reason, at most you see a higher variance.

Sometime I saw the tg values are not consistent from run to run (low variance). Thus sometime I need to run several times to verify the change

With your latest results you didn't even show whether you used FORCE_MMVQ or not.

I am not using flags in the above benchmark result

Regardless, a blanket disable on UMA devices makes no sense

I tested the benchmark using k=28672 (Llama 3.1 70B) and 53248 (Llama 3.1 405B)

Due to the lack of a dedicated Q6_K MMVQ compute shader, current Q6_K MMVQ performance falls behind Q2_K MMVQ, Q4_K MMVQ, and non-MMVQ Q6_K execution path. I believe that implementing a dedicated Q6_K MMVQ shader would close this gap, bringing the performance of the MMVQ-enabled and disabled paths very close to parity. Additionally, performance regressions become noticeable on the Q8_0 path once the matrix inner dimension hits $K = 53,248$.

Therefore I would suggest adding the max k to 28672 to avoid any regression

you should really not be using claude to try to explain the reason for performance behaviour, it is guaranteed to hallucinate

Agree. Further cross-check is needed. Also I am not sure if it is hallucinate but after searching I guess the suggestion comes from this discussion

But AI models write better than me that is no doubt